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The purpose of this paper is to review common approaches to performance evaiuation in accountabiiity 
systems, in order to recommend a workabie approach for Caiifornia Community coiieges, as the system 
seeks to meet the requirements of district-ievei accountabiiity required by AB 1417. The recommendations 
within this paper capture commoniy heid views about the most appropriate approach to evaiuation heid by 
members of the foiiowing groups: 

• The RP/CSS Review Panei, charged with deveioping a performance framework and indicators for 
discussion and review reiated to the impiementation of AB 1417 

• Members of a Panel of Externai Experts, charged with reviewing and recommending changes to 
the work of the RP/CSS 

• The Statewide Oversight Committee of CA Community Coiiege Leaders, charged with reviewing 
and recommending changes to the work of the RP/CSS 

Common Approaches to Performance Evaluation 

Approaches to performance evaluation in accountability systems have been varied. Some of the most 
common approaches include: (1) some version of comparing institutions to each other within a system; (2) 
comparing institutions to a common standard or benchmark; and (3) an improvement approach, which looks 
at individual institutional performance as it changes over time, as initiatives are undertaken to improve 
student performance. Additional approaches are usually some combination of the three. 

Through versions of comparing institutions (1), aggregate student achievement measures are compared 
to each other at the institutional level and/or to a statewide “average.” When using averages, many 
institutions will be above the average, but large numbers will also be below it. Moreover, by definition, the 
basic underlying assumption to comparing institutions is that some institutions are doing well in serving their 
students and some are not. By their very nature, such systems dictate that there will be winners and there 
will be losers, regardless of the actual level of performance that is achieved for each measure. Such 
systems are adjusted from time to time, by considering exogenous factors, an attempt to “level the playing 
field” or by creating peer groups for analysis, as another means of leveling the playing field. Because so 
many of the differences among colleges are due to factors beyond the control of the colleges, such 
performance evaluation systems rarely achieve sufficient fairness. Performance evaluation systems that 
rely on comparisons of institutions fail to capture the full environmental complexity in which a college serves 
its students, e.g., local demographic shifts, state of the local economy, varying political cultural contexts, 
varying educational levels of students enrolling and seeking services, varying student goals, institutional 
governance, institutional finances subject to state budget constraints, and the like. 

With versions that compare institutions to a common standard or benchmark (2), aggregate student 
performance in institutions is evaluated relative to an established standard or benchmark, either by 
specifying scores as standards to be met, or specifying general categories of scores, such as “advanced,” 
“proficient,” and “below proficient.” Although this approach looks to establish an acceptable level of 
performance, it too falls short, due to the same varying levels of complexities facing the colleges that are 
described above. As pointed out by a member of the External Review Panel who has expertise and 
experience in performance evaluation systems, deciding what the standard should be in such systems is a 
“daunting task.” Much of this is due to this varying levels of complexity. For example, how can a standard 
be comparable for a community college that serves primarily an urban area with low socioeconomic and 
educational levels to those of a suburban community college where socioeconomic levels are high, K-12 
resources are high and students are generally supported throughout their education by educated parents? 

With the institutional student improvement approach, sometimes referred to as “value-added” (3), 

individual institutional performance changes are tracked from year to year by looking at the environment 
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(i.e., the inputs, such as student abilities upon entry and the various other environmental inputs, mentioned 
above) to develop interventions and assessment of progress over time. This approach mitigates the 
problems of the previously described methods because the institution is compared with itself within the 
context of the complex environment in which it operates, an environment that continually evolves over time. 
The value of this approach is that it is student-centered, focusing on where students start and how they 
progress. This approach includes a review of interventions that are implemented to help students improve 
their performance. An additional important strength of this approach is that it can account for achievement 
reflected both by self-defined student goals (that vary across districts), as well as by statewide goals. For 
example, immigrant students, who first come to colleges with a goal to improve their English speaking, may 
achieve their goal and leave; the college experience has assisted these students to be successes, in terms 
of their individually defined goals. These students indeed improve, despite the fact that the improvement 
does not contribute toward the overall institutional performance objective that has been designated in a 
statewide system. Nonetheless, through the institutional student improvement approach, the goal 
achievement of these immigrant students can and should be documented by districts. The value of this 
performance approach is that institutions pause, look at student performance changes over time, undertake 
their own self-evaluation to understand those changes within the institution’s environmental context, and 
implement programs and services for student performance improvement. The key here is the issue of self- 
evaluation for continuous improvement. This approach is used by many states, including Washington, 
Tennessee, Missouri, Ohio, Kentucky, Minnesota, and Arkansas. (Chapter 5, Performance Funding For 
Public Fliqher Education: Fad or Trend? New Directions for Institutional Research Burke and Serban, 1998) 

An underlying premise for the following proposal is based upon the fact that there is a strong culture of 
institutional self-assessment, which has grown from the grass roots level in California, as faculty strive to 
find new ways to help students learn. This culture of institutional self-assessment has been greatly 
enhanced by accreditation processes and the state’s accountability system, both of which require faculty 
and managers, not only to find new ways to help students, but also to document, evaluate and alter these 
new initiatives, as appropriate. 



Recommended Approach for California 

The California community college performance evauation system, developed to meet the requirements of 
AB 1417, will contribute to the already rich array of self-assessments and external evaluations that assist 
colleges to measure their effectiveness, develop new programs and services to help students succeed, and 
improve teaching, learning and the prudent use of resources. In interpreting the meaning of the perfomance 
indicator data, it is crucial to recognize that districts and their colleges operate within diverse environments 
and cultures, and it is these attributes that shape opportunities, challenges, advantages and disadvantages 
to promote and have students achieve. 

The new performance system will be of paramount value to community college constituencies, particularly 
to students, if the system emphasizes the use of the data by the districts to examine themselves. While 
there may be a temptation to rank institutions based on the performance indicator data, doing so is likely to 
divert attention away from improvement. As pointed out by External Panel of Experts member, Trudy Bers, 
“Institutions that are ranked lower will feel challenged to explain themselves and institutions ranked higher 
will succomb to self satisfaction, while consideration of the diversity of institutions and their environments is 
lost.” 

External Panel of Experts member, Peter Ewell makes a further point, by suggesting that not only is ranking 
inappropriate, it may encourage districts to serve constiutiences that will improve performance, rather than 
meeting real educational needs in the local community: ““You are certainly right to reject rankings-and in 
fact (reject) any approach that will try to distinguish levels of institutional performance based on fine cut 
criteria and in areas where student clientele is the major factor. In fact, since student characteristics are the 
major driver in almost all of these measures, you can make the further point that adopting rankings would 
induce institutions *not* to serve their public missions by seeking a different clientele. “ 
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And finally, External Panel of Experts member Joseph Burke suggests that the institutional student 
improvement model, is the best evaluation to follow: “I strongly endorse the institutional improvement 
approach as the one that best fits student development and district needs, which vary tremendously among 
California community college districts. The improvement approach puts the accent where it belongs on the 
student and the district and not on institutional comparisons, which are misleading.” 

National experience with ranking systems has been controversial, evidence that rankings have let to 
institutional improvement is sparse and consensus about the value of rankings does not exist. As such, 
introducing rankings into the new AB 1417 System is unlikely to have benefits and is very likely to create 
controversy and misunderstandings that will divert attention away from the important goal of improvement. 

California has a broad culture of institutional self-assessment that already exists in its districts, driven by 
internal mechanisms that have been developed, as well as external review requirements. Among the many 
systems currently in place are: 

• College and district level institutional effectiveness reporting structures 

• Program review processes that involve internal reviews as well as external reviews by local business 
advisory groups, by professional peer groups and external accrediting agencies 

• Faculty-initiated inquiry and training, dedicated to instructional improvement in the classroom (known 
as flex activities and workshops) 

• District and college level comprehensive self-assessment through the external institutional 
accreditation process as well as through the external program accreditation processes mentioned 
above 

• Accountability to the U.S. Dept, of Education related to Vocational Technical Education Act and the 
Integrated Postsecondary Education System requirements 

• Matriculation and Student Equity review processes 

• The new learning assessment reporting requirements instituted by the regional accreditating board 
for junior and community colleges in all western states (ACCJC), and finally, 

• The Partnership for Excellence project initiated in the late 1990s (soon to be replaced by the AB 
1417 system) 

These systems currently in place in all community college districts of California demonstrate a strong culture 
of self-assessment. Likewise, a great deal of systemwide evaluation is conducted by the State Chancellor’s 
Office. Therefore, it is the recommendation of the RP/CSS Review Panel (with the endorsement of 
members of the External Panel of Experts and the CA Community College Oversight Committee) that the 
state of California utilize the existing district and state level infrastructure for performance evaluation within 
the AB 1417 System. Specific to this are the following recommendations for evaluation, once initial 
implementation of AB 1417 takes place and measures are generated: 

1. Once generated. District Core and State Level Core measures should be sent to the districts for 
review to prepare an evaluation of the data within the local context. This evaluation should be 
discussed broadly across the district and with the Board of Trustees, culminating in a report that is 
sent to the State Chancellor’s Office that includes: 

■ A brief review of any change in district core measures over time {minimum of three years) 

■ A brief discussion of why district core measures are important in the local context 

■ A brief description of limitations and/or advantages beyond local district control that impact 
district core measures 

■ A brief description of specific action plans for improvement, as appropriate 

2. The State Chancellor’s Office should provide an overview/summary report to the Board of 
Governors, Department of Finance and the Legislative Analyst Office of the District Core, State 
Level Core and themes gleaned from the district reports. 
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Upon Gaining Experience with the AB 1417 System: 

After a few years of implementation, the districts and the State Chancellor’s Office will have a better 
understanding of the performance measures and how well performance evaluation is functioning to help 
students succeed. This review should include an evaluation of the effectiveness of the actual measures 
themselves in terms of whether they provide districts with good information for institutional student 
improvement. 
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